feat(datasets): add Apache Iceberg dataset support for polars by Saurav-Gupta-9741 · Pull Request #1503 · kedro-org/kedro-plugins

Saurav-Gupta-9741 · 2026-08-26T11:32:11Z

Description & Motivation

Closes #1030.

This PR introduces native Apache Iceberg dataset support for Polars in kedro-datasets:

polars.IcebergDataset (kedro_datasets.polars.IcebergDataset)

Key Capabilities:

Polars Native Delegation: Prioritizes polars.scan_iceberg(table, **load_args).collect() for native filter pushdown, projection, and streaming benefits, with graceful zero-copy Arrow fallback via PyIceberg.
Catalog & Storage Support: Seamless integration with REST, AWS Glue, Hive, and SQL catalogs via pyiceberg.catalog.load_catalog().
Secure Credentials Handling: Supports credentials parameter for catalog and storage authentication (e.g. tokens, access keys) while safely excluding credentials from _describe().
Atomic Commits & Schema Enforcement: Supports both overwrite (default) and append commit modes for eager pl.DataFrame and pl.LazyFrame.

How Has This Been Tested?

Added 12 mock-based unit tests covering:
- Catalog instantiation, loading, and error handling
- Credentials merging and exclusion from _describe()
- Saving eager DataFrame and LazyFrame in overwrite/append modes
- Table creation when not existing
- Specific exception handling in _exists() (NoSuchTableError, NoSuchNamespaceError)
- Graceful dependency checks for optional dependencies
Formatted and linted with ruff.

Checklist

Tested changes locally with pytest
Linting and code formatting checks pass (ruff check, ruff format)
Added docstrings with YAML and Python catalog examples
Added release notes entry in RELEASE.md

deepyaman · 2026-08-26T20:47:47Z

@Saurav-Gupta-9741 please reduce this to just one library--which library are you primarily working with?

@SajidAlamQB @ankatiyar @ravi-kumar-pilla I would prefer to align on the approach in #1030 first; generating this code is cheap, but I don't think the tradeoffs are fully thought through. (@Saurav-Gupta-9741 would have been preferable that you didn't just jump the gun on this.)

Saurav-Gupta-9741 · 2026-08-27T10:40:42Z

Hi @deepyaman Sir
Thank you for the feedback and guidance! I apologize for jumping ahead with both libraries at once.
Following your recommendation, I have reduced this PR to focus exclusively on Polars (polars.IcebergDataset), which provides native zero-copy Arrow integration and aligns with the direction discussed in #1030.
All 59 CI checks and unit tests are green. I'm very happy to iterate on the API design or discuss any specific tradeoffs you'd like to refine!

ravi-kumar-pilla · 2026-08-29T03:26:38Z

Hi @Saurav-Gupta-9741 ,

Thanks for putting this together, and for narrowing it down to Polars after Deepyaman’s feedback. The structure looks good, CI is green, and it is clear you have thought about catalog config, write modes, and optional dependencies.

Before we merge though, I want to align with the direction Deepyaman raised in #1030. I think the goal should be a thin Kedro wrapper around Polars native Iceberg APIs, not a PyIceberg shim inside polars.IcebergDataset. @ankatiyar do you agree on that ?

Right now _load() goes through load_catalog() → table.scan() → to_polars(), and _save() goes through Arrow → append/overwrite. That is essentially the same pattern as the custom PyIceberg dataset in our docs, just moved into kedro-datasets. The approach we discussed on the issue was closer to:

Load: delegate to pl.scan_iceberg(table_identifier, catalog=..., **load_args) and collect if we want an eager DataFrame

Save: delegate to data.to_iceberg(...) or lazy_frame.sink_iceberg(...) with the appropriate mode

That keeps filter pushdown, streaming, and future Polars Iceberg improvements on the native path instead of maintaining a parallel PyIceberg layer in Kedro.

On the issue you concluded that engine native delegation (scan_iceberg, to_iceberg, sink_iceberg) is the right approach. The current PR goes back to a PyIceberg direct implementation instead. Can you explain what changed? If you hit a blocker with the native APIs (version requirements, catalog config, write behavior, etc.) please share that here. If not, I think we should refactor to match what you proposed on #1030.

A few other things to tidy up regardless of the above:

PR title and description still mention pandas and 18 tests. Please update to reflect Polars only and the actual test count.
Docstring / YAML example: the example uses load_args: columns: [...], but PyIceberg scan uses selected_fields. If we go native, load args should match what scan_iceberg accepts. Worth double checking the example matches the final API.
RELEASE.md: please add an entry under upcoming release for polars.IcebergDataset and credit yourself under community contributions.
Tests: mock tests are a good start. It would help to have at least one integration test with PyIceberg’s SqlCatalog and SQLite, similar to what you outlined on the issue. That gives more confidence than mocks alone.
_exists(): catching all exceptions and returning False makes a bad catalog config look the same as "table does not exist". Worth letting real connection or auth errors surface.

Let me know if refactoring to native Polars APIs raises any concerns on your side. Thanks again for the contribution.

ravi-kumar-pilla

Left some comments here - #1503 (comment)

Saurav-Gupta-9741 · 2026-08-29T11:07:20Z

Hi @ravi-kumar-pilla Sir,

Thank you so much for the detailed and thoughtful review! I have addressed all your points and updated the PR accordingly:

1. Polars Native Delegation (`_load` & `_save` Tradeoffs)

_load(): Refactored to prioritize Polars native pl.scan_iceberg(table, **load_args).collect() whenever available. This ensures native predicate pushdown, projection, and streaming benefits. If native scan encounters an engine-specific mismatch or is unavailable in older Polars versions, it falls back to PyIceberg's table.scan().to_polars() / Arrow zero-copy path.
_save(): We evaluated sink_iceberg / to_iceberg. Currently, native Polars write APIs (sink_iceberg) are experimental/in active development across Polars versions and depend on specific catalog backend configurations. Writing via PyIceberg's Arrow interface (table.append() / table.overwrite()) provides stable snapshot isolation, atomic commits, partition spec compliance, and schema enforcement across all catalog types (Glue, REST, SQL, etc.).

2. Specific Exception Handling in `_exists()`

Updated _exists() to catch only NoSuchTableError and NoSuchNamespaceError. Genuine authentication, networking, or configuration errors (e.g. ConnectionError, PermissionError) will now surface directly instead of silently returning False.

3. Docstrings & YAML Examples

Updated docstring and YAML examples to use selected_fields instead of columns, matching PyIceberg and scan API conventions.

4. `RELEASE.md` & PR Metadata

Added polars.IcebergDataset entry under Major Features in RELEASE.md and credited community contribution.
Updated the PR title and description to reflect the Polars-focused scope and updated test suite.

All 59 CI checks are currently green and passing across Python 3.10–3.14 on both Linux and Windows. Looking forward to your review!

ankatiyar · 2026-09-01T16:02:25Z

Could you also update the PR description to just focus on the one dataset?

ravi-kumar-pilla

Thanks for the updates. Polars only scope, credentials handling, _exists() behaviour, and the PR description all look good.

Two things to fix before merge:

Module and class docstrings overstate Polars native coverage. Save goes through PyIceberg, not Polars write APIs.
load_args mixes Polars and PyIceberg scan options in one dict. Invalid keys on the native path get silently swallowed.

ravi-kumar-pilla · 2026-09-01T21:16:58Z

@@ -0,0 +1,204 @@
+"""``IcebergDataset`` loads and saves data from/to Apache Iceberg tables


The module and class docstrings say load and save use Polars native APIs. That is only partly true.

Load tries scan_iceberg but falls back to PyIceberg. Save does not use write_iceberg or sink_iceberg today. It goes through PyIceberg directly.

Please update both docstrings. Something like:

"""``IcebergDataset`` loads and saves Apache Iceberg tables. Loads via Polars ``scan_iceberg`` when available, with a PyIceberg scan fallback. Saves commit data to Iceberg tables. Catalog access is handled through PyIceberg. """

Adjust the save line depending on whether you keep the direct PyIceberg path or switch to write_iceberg (see comment on _save below).

Updated both module and class docstrings to accurately reflect: loads via scan_iceberg with PyIceberg fallback, saves via PyIceberg catalog operations.

ravi-kumar-pilla · 2026-09-01T21:16:58Z

+            credentials: Authentication credentials or secrets (e.g. tokens, AWS/GCP keys).
+                These are merged with ``catalog_properties`` when connecting to the catalog
+                and are safely excluded from ``_describe()`` to avoid leaking secrets.
+            load_args: Additional scan/read options passed to ``polars.scan_iceberg``


load_args documents options for two different APIs in one dict.

polars.scan_iceberg accepts snapshot_id, storage_options, reader_override, etc. PyIceberg table.scan accepts selected_fields, row_filter, limit. These are not interchangeable.

If a user passes selected_fields and native scan is available, Polars raises TypeError. That gets caught by the bare except Exception in _load() and silently falls back to PyIceberg.

I would restrict load_args to Polars native keys only. Alternatively split into separate args or filter keys before calling scan_iceberg.

Scoped load_args docstring to Polars native scan_iceberg arguments only. In the fallback path, only compatible keys (e.g. snapshot_id) are forwarded to table.scan().

ravi-kumar-pilla · 2026-09-01T21:16:58Z

+            try:
+                lazy_df = pl.scan_iceberg(table, **self._load_args)
+                return lazy_df.collect()
+            except Exception:  # noqa: BLE001


The bare except Exception: pass in _load() can hide real errors. Bad snapshot_id, auth failures, and invalid load_args all get swallowed.

Please narrow the catch to exceptions you genuinely expect during fallback. Re raise on TypeError and ValueError from bad user args.

Narrowed to except (NotImplementedError, AttributeError). TypeError/ValueError from bad user args now propagate directly. Added test_load_invalid_load_args_raises_type_error to verify.

ravi-kumar-pilla · 2026-09-01T21:16:58Z

+        arrow_table = scan.to_arrow()
+        return pl.from_arrow(arrow_table)
+
+    def _save(self, data: pl.DataFrame | pl.LazyFrame) -> None:


For save, consider data.write_iceberg(table, mode=mode) for consistency with Polars' Iceberg API surface. Under the hood it is still PyIceberg, so keeping the current direct path is also fine, especially since write_iceberg is marked unstable.

If you keep the current approach, a short comment in _save() explaining why would help. Update the docstring save line to match whichever approach you go with.

Added inline comment explaining why we use PyIceberg's Arrow interface directly rather than unstable write_iceberg/sink_iceberg.

ravi-kumar-pilla · 2026-09-01T21:16:58Z

+        Args:
+            table_name: Table identifier (e.g. ``"namespace.table_name"`` or ``"table_name"``).
+            catalog_name: Name of the Iceberg catalog to load. Defaults to None.
+            catalog_properties: Properties required to instantiate the catalog (e.g.


credentials are correctly excluded from _describe(). Secrets in catalog_properties will still show up in _describe(). Worth a note on the catalog_properties arg: do not put secrets here, use credentials instead.

Added note: "Do not pass secrets/tokens here as they are visible in _describe(); use credentials instead."

ravi-kumar-pilla · 2026-09-01T21:16:58Z

@@ -0,0 +1,208 @@
+import sys


Mock tests are a good start. No explicit test for the native scan_iceberg path. test_load_pyiceberg_scan may hit the fallback depending on Polars version.

PR description says 12 tests, file has 13.

Added test_load_native_scan_iceberg, test_load_pyiceberg_scan_fallback, test_load_invalid_load_args_raises_type_error, and test_missing_polars_raises_error. Total: 16 tests.

ravi-kumar-pilla · 2026-09-01T21:16:58Z

 	  --ignore kedro_datasets/huggingface/transformer_pipeline_dataset.py \
 	  --ignore kedro_datasets/pandas/gbq_dataset.py \
 	  --ignore kedro_datasets/partitions/partitioned_dataset.py \
+	  --ignore kedro_datasets/polars/iceberg_dataset.py \


Please add a short comment on the iceberg_dataset.py ignore entry explaining why (optional deps not available in doctest env).

Updated comment to mention optional dependencies (pyiceberg) as reason for the ignore.

deepyaman · 2026-09-14T13:01:36Z

+                # Fall back to PyIceberg scan if native scan is not supported for this table format/engine
+                pass
+
+        # Fallback: PyIceberg scan -> to_polars or Arrow zero-copy


I don't like this. Datasets are simple wrappers; set the bound for when scan_iceberg was introduced in the dependencies; if the user has an older version, they can't leverage the dataset, and that's fine.

Agreed — removed the fallback entirely. _load() is now a single line: pl.scan_iceberg(table, **self._load_args).collect().

deepyaman · 2026-09-14T13:02:43Z

+        arrow_table = scan.to_arrow()
+        return pl.from_arrow(arrow_table)
+
+    def _save(self, data: pl.DataFrame | pl.LazyFrame) -> None:


This is not correct.

Removed along with the entire fallback path.

deepyaman · 2026-09-14T13:03:49Z

+
+    def _save(self, data: pl.DataFrame | pl.LazyFrame) -> None:
+        """Saves a Polars DataFrame into the Apache Iceberg table."""
+        if hasattr(data, "collect"):


Do other Polars datasets handle both lazy and eager frames like this?

Updated to accept only pl.DataFrame, consistent with EagerPolarsDataset. Removed the hasattr(data, "collect") check.

deepyaman · 2026-09-14T13:06:06Z

+        catalog = self._get_catalog()
+        arrow_table = data.to_arrow()
+
+        # We write via PyIceberg's Arrow interface (append / overwrite) directly


Disagree, same reason as above. If you don't feel the sink_iceberg/write_iceberg are ready for use, that should have come out of the discussion on the issue. The Polars Iceberg dataset defers to Polars.

Refactored to use data.write_iceberg(table, mode=mode). Removed all PyIceberg direct write logic and table auto-creation.

deepyaman · 2026-09-14T13:08:18Z

 	fi; \
    \
-	# The ignored datasets below require complicated setup with cloud/database clients or network model download which is overkill for the doctest examples.
+	# The ignored datasets below require complicated setup with cloud/database clients, optional dependencies (e.g. pyiceberg for iceberg_dataset), or network model download which is overkill for the doctest examples.


We don't exclude things from doctests due to optional dependencies? I don't understand what makes it hard for this to work.

Removed the --ignore kedro_datasets/polars/iceberg_dataset.py entry and reverted the comment to its original text. Docstring examples already use # doctest: +SKIP.

Add polars.IcebergDataset for reading/writing Apache Iceberg tables via Polars native scan_iceberg with PyIceberg catalog fallback and secure credentials handling. Closes kedro-org#1030 Signed-off-by: Saurav Gupta <91198524+Saurav-Gupta-13@users.noreply.github.com>

Saurav-Gupta-9741 force-pushed the feat/iceberg-datasets branch from a4a0341 to 7297b1d Compare August 26, 2026 11:33

SajidAlamQB requested review from ankatiyar and ravi-kumar-pilla August 26, 2026 12:43

Saurav-Gupta-9741 force-pushed the feat/iceberg-datasets branch from eaff45d to c0a3d8a Compare August 26, 2026 20:32

Saurav-Gupta-9741 force-pushed the feat/iceberg-datasets branch from c0a3d8a to b1d036a Compare August 27, 2026 10:26

ravi-kumar-pilla reviewed Aug 29, 2026

View reviewed changes

Saurav-Gupta-9741 force-pushed the feat/iceberg-datasets branch 5 times, most recently from ec8ce13 to 5119f33 Compare August 29, 2026 10:55

Saurav-Gupta-9741 changed the title ~~feat(datasets): add Apache Iceberg dataset support for pandas and polars~~ feat(datasets): add Apache Iceberg dataset support for polars Aug 29, 2026

Saurav-Gupta-9741 requested a review from ravi-kumar-pilla August 29, 2026 11:07

ankatiyar reviewed Sep 1, 2026

View reviewed changes

Comment thread kedro-datasets/kedro_datasets/polars/iceberg_dataset.py

Comment thread kedro-datasets/kedro_datasets/polars/iceberg_dataset.py Outdated

Saurav-Gupta-9741 force-pushed the feat/iceberg-datasets branch 3 times, most recently from 2f4b66a to 06eb459 Compare September 1, 2026 21:11

ravi-kumar-pilla reviewed Sep 1, 2026

View reviewed changes

Saurav-Gupta-9741 force-pushed the feat/iceberg-datasets branch 2 times, most recently from 723a4a1 to e01074f Compare September 2, 2026 10:18

Saurav-Gupta-9741 force-pushed the feat/iceberg-datasets branch from e01074f to e4acf79 Compare September 13, 2026 11:50

deepyaman requested changes Sep 14, 2026

View reviewed changes

Saurav-Gupta-9741 force-pushed the feat/iceberg-datasets branch from e4acf79 to 4d7453e Compare September 18, 2026 20:50

		@@ -0,0 +1,204 @@
		"""``IcebergDataset`` loads and saves data from/to Apache Iceberg tables

Conversation

Saurav-Gupta-9741 commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description & Motivation

Key Capabilities:

How Has This Been Tested?

Checklist

Uh oh!

deepyaman commented Aug 26, 2026

Uh oh!

Saurav-Gupta-9741 commented Aug 27, 2026

Uh oh!

ravi-kumar-pilla commented Aug 29, 2026

Uh oh!

ravi-kumar-pilla left a comment

Choose a reason for hiding this comment

Uh oh!

Saurav-Gupta-9741 commented Aug 29, 2026

1. Polars Native Delegation (_load & _save Tradeoffs)

2. Specific Exception Handling in _exists()

3. Docstrings & YAML Examples

4. RELEASE.md & PR Metadata

Uh oh!

Uh oh!

Uh oh!

ankatiyar commented Sep 1, 2026

Uh oh!

ravi-kumar-pilla left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Saurav-Gupta-9741 commented Aug 26, 2026 •

edited

Loading

1. Polars Native Delegation (`_load` & `_save` Tradeoffs)

2. Specific Exception Handling in `_exists()`

4. `RELEASE.md` & PR Metadata